Skip to content

Add native LLM core foundation#24712

Merged
kitlangton merged 199 commits intodevfrom
llm-core-patch-api
May 8, 2026
Merged

Add native LLM core foundation#24712
kitlangton merged 199 commits intodevfrom
llm-core-patch-api

Conversation

@kitlangton
Copy link
Copy Markdown
Contributor

@kitlangton kitlangton commented Apr 28, 2026

Summary

  • Add packages/llm, a native Effect-based LLM core with typed request/event schemas, route/protocol/provider composition, tool runtime, structured object generation, and recorded provider golden tests.
  • Add @opencode-ai/http-recorder, a private workspace test helper for deterministic HTTP/WebSocket cassette recording and replay used by the LLM package.
  • Add provider coverage for OpenAI, Anthropic, Gemini, Bedrock Converse, OpenAI-compatible providers, OpenRouter, xAI, GitHub Copilot, Azure, and Cloudflare Workers AI / AI Gateway.

Safety

  • PR scope is limited to the standalone LLM package, HTTP recorder test helper, and minimal workspace wiring.
  • OpenCode runtime bridge/integration code and the AI SDK migration design doc were removed from this PR so the package can merge independently.
  • Recorded cassettes redact provider credentials and Cloudflare account/gateway identifiers; auth headers are not stored.

Testing

  • cd packages/llm && bun typecheck
  • cd packages/llm && bun run test
  • cd packages/http-recorder && bun typecheck
  • cd packages/http-recorder && bun test

kitlangton added 25 commits May 1, 2026 08:11
- Structurally match recorded requests by canonical JSON so non-deterministic
  field ordering doesn't break replay.
- Pluggable header allow-list and body redaction hook on the record/replay
  layer, so adapters with non-default auth (Anthropic, Bedrock) can plug in
  without touching this file.
- Move the cassette-name dedupe set inside recordedTests() so two describe
  files using different prefixes can run in parallel.
- Replace inline SSE template literals and per-file HTTP layers with shared
  test/lib helpers (sseEvents, fixedResponse, dynamicResponse, truncatedStream).
- Tighten recorded-test assertions to exact text and usage so adapter parser
  regressions surface immediately instead of passing fuzzy length>0 checks.
- Add cancellation and mid-stream transport-error tests for the OpenAI Chat
  adapter.
- Add cross-phase patch tests that verify each phase sees an updated
  PatchContext and that same-order patches sort deterministically by id.
- shared sse helper now expects Effectful decodeChunk and process callbacks,
  so adapter parsers can be Effect.gen and yield typed ProviderChunkError
  instead of throwing across the sync mapAccum boundary.
- parseJson returns Effect<unknown, ProviderChunkError> via Effect.try,
  matching the package style guide on yieldable errors.
- OpenAI Chat finalizes accumulated tool inputs eagerly when finish_reason
  arrives, surfacing JSON parse failures at the boundary instead of at halt.
  onHalt stays sync and just emits from state.
- generate's runFold reducer now mutates the accumulator instead of
  reallocating the events array on every chunk, dropping O(n^2) growth on
  long streams.
Gemini rejects integer enums, dangling required fields, untyped arrays, and
object keywords on scalar schemas. The sanitizer was previously a divergent
copy in OpenCode; this lands it in the package as a tool-schema patch with
deterministic tests and selects it for Gemini-protocol or Gemini-named models.

Also tightens the Gemini test suite: covers tool-choice none, drops the
tool-input-delta assertion that Gemini does not actually emit, and confirms
total usage stays undefined when only thoughtsTokenCount arrives.
…integration

Updates the AGENTS.md TODO list:
- mark Responses, Anthropic, and Gemini adapter coverage as done
- mark the Gemini schema sanitizer port as done
- add concrete next-step items for OpenCode integration: ModelRef bridge,
  request bridge, provider-quirk patches, request/stream parity tests, and
  a flagged rollout against existing session/llm.test.ts cases
- add OpenAI-compatible Chat, Bedrock Converse, and Vertex routing as
  outstanding adapter/dispatch decisions
Every adapter's parse already produces LLMEvents (via the process callback in
the shared sse helper), and every raise was Stream.make(event). The Chunk type
parameter, the raise field, the RaiseState interface, and the Stream.flatMap
raise step in client.stream were all pure overhead.

- Adapter contract shrinks from <Draft, Target, Chunk> to <Draft, Target>.
- All four adapters drop their raise: (event) => Stream.make(event) line.
- client.stream skips the no-op flatMap.
- AGENTS.md adapter section reflects the simpler contract.
Per the package style guide, sync if/return functions that need to fail
should yield the error directly via Effect.gen rather than ladder
Effect.fail / Effect.succeed across every branch.

Touches all four adapters' tool-choice lowering. The naming-required
validation now reads as 'guard, then return' rather than embedded in a
chain of monadic returns. Behavior unchanged.
Locks down the error contract before OpenCode integration:
- mid-stream provider errors (Anthropic 'event: error', OpenAI Responses
  'type: error') surface as 'provider-error' LLMEvents
- HTTP 4xx responses fail with ProviderRequestError before stream parsing
  begins (the executor contract)

Anthropic already had both. Adds:
- OpenAI Responses: provider-error fixture, code-fallback fixture, HTTP 400
- OpenAI Chat: HTTP 400 sad path
- AGENTS.md TODO refreshed; live recordings of provider errors still pending
Schema-first, Effect-first tool loop:

- 'tool({ description, parameters, success, execute })' constructs a fully
  typed Tool. parameters and success are Effect Schemas; execute is typed
  against them and returns Effect<Success, ToolFailure>. Handler dependencies
  are closed over at construction time so the runtime never sees per-tool
  services.
- 'ToolRuntime.run(client, { request, tools, maxSteps?, stopWhen? })' streams
  the model, decodes tool-call inputs against parameters, dispatches to the
  matching handler, encodes results against success, emits tool-result events,
  appends assistant + tool messages, and re-streams. Stops on non-tool-calls
  finish, maxSteps, or stopWhen.
- Three recoverable error paths emit tool-error events so the model can
  self-correct: unknown tool name, input fails parameters Schema, handler
  returns ToolFailure. Defects fail the stream.
- 'ToolFailure' added to the schema and exported as the single forced error
  channel for handlers.
- Tool definitions on the LLMRequest are derived via toJsonSchemaDocument so
  consumers don't write JSON Schema by hand.

8 deterministic fixture tests cover the loop, errors, maxSteps, stopWhen, and
parallel tool calls in one step.
kitlangton added 11 commits May 7, 2026 09:09
Three small fixes for stale references after the recent
adapter→route rename and `model.protocol` field removal in
@opencode-ai/llm:

- Update three `@opencode-ai/llm/adapter` imports to `/route`.
- Switch `NATIVE_PROTOCOLS` filter on `model.protocol` to
  `NATIVE_ROUTES` on `model.route`.
- Refresh test fixtures: `protocol:` → `route:`, drop `apiKey:`
  / `adapter:` assertions that reference fields no longer on
  `ModelRef`, drop the `attachments` expectation that no longer
  has a source-side handler.
- Default `key: "test-key"` on `ProviderTest.info` so prepare()
  can resolve auth without per-test env setup.
Push URL host knowledge from `Endpoint` (route layer) up to `model.baseURL`
(provider helper layer). The route just composes a path onto whatever
host the model already carries.

Endpoint:
- `Endpoint<Body>` shrinks to `{ path }`. No more `default`, `baseURL`,
  or `required` fallback fields.
- `Endpoint.path(value)` replaces the old `Endpoint.baseURL({...})` factory.
- `Endpoint.render()` is now sync — `${model.baseURL}${path}` plus query
  params. No fallback chain, no Effect wrapper.

ModelRef:
- `ModelRef.baseURL: Schema.String` (was optional). Every materialized
  model carries a host.
- `RouteModelInput.baseURL?: string` stays optional so route defaults
  can supply a canonical URL; routes without a default tighten it.

Provider helpers:
- Each protocol exports a `DEFAULT_BASE_URL` constant and bakes it into
  `route.defaults.baseURL`. Provider helpers don't need to set baseURL.
- Azure uses a new `AtLeastOne<T>` helper from `auth-options.ts` to
  require either `resourceName` or `baseURL` at the type level.
- Bedrock provider computes baseURL from `region` at construction time.
- OpenAI-compatible profiles now have required (not optional) `baseURL`
  in their type — all 9 already supplied one.
- The `defaultBaseURL: string | false` knob on protocol endpoint
  factories is gone.

Effects:
- Forgetting baseURL is now caught at compile time (TS) or model
  construction time (modelWithDefaults runtime check), not request time.
- `Endpoint.render` no longer needs Effect wrapping in the transport
  hot path.
Bring the package guide in line with everything that's landed (route
rename, body/event vocabulary, schema split, endpoint simplification,
WebSocket transport, AuthOptions.bearer, dispatched protocol step).

- Update AGENTS.md throughout: payload→body, chunk→event, processChunk→step,
  Endpoint.baseURL({...})→Endpoint.path(...), refreshed folder layout,
  refreshed Routes / URL Construction / Provider Definitions sections.
- Fold HOUSE_STYLE.md (protocol file shape, rules, review checklist) into
  AGENTS.md as a "Protocol File Style" section.
- Delete the four DESIGN.*.md proposals that are fully implemented:
  routes-protocol-transport, websocket-transport, http-retry, model-options.
- Delete TOUR.md — it had grown into a 700-line narrative walkthrough that
  duplicated AGENTS.md with stale vocabulary. example/tutorial.ts is the
  canonical reading path now.
Three small cleanups in the OpenAI Responses protocol:

- Unify `HOSTED_TOOL_NAMES` + `hostedToolInput` into one `HOSTED_TOOLS`
  record per tool: `{ name, input: (item) => unknown }`. Adding a new
  hosted tool is now a single entry instead of two parallel switches that
  must stay in sync.
- Tighten `isHostedToolItem`'s narrowing to include the `type` field, so
  callers know they're dealing with a known hosted-tool shape (not just
  "has an id"). Drives a cleaner `hostedToolEvents` signature.
- Split the body schema into a shared `OpenAIResponsesCoreFields` record
  used by both the HTTP body (adds `stream: true`) and the WebSocket
  `response.create` message (adds `type`). Removes the destructure-and-
  strip dance at schema definition time. Runtime conversion in
  `webSocketMessage` still strips `stream` because OpenAI's WebSocket API
  doesn't expect it on the wire.

Plus a tiny fix in bedrock-converse.ts: explicit `Route.model<BedrockConverseModelInput>`
type argument so the mapInput overload selects properly (was inferring
to the narrower `RouteModelInput`).
Capture the assessment of how opencode integrates the AI SDK today and
the phased plan to replace it with @opencode-ai/llm behind a feature flag.

Sections:
- Today's architecture, including the trace of one streamText call and
  the existing native path's gate conditions
- Where the spaghetti actually lives (AI SDK type leakage in 11+ files,
  scattered provider-specific transforms, the provider/sdk/copilot/* fork,
  duplicated MessageV2 conversion)
- Target architecture (one flag, one decision point at layer construction)
- Phased migration: Decouple → Service swap → Native parity → Flag rollout
  → Delete AI SDK
- Suggested execution order, key files, risks and open questions
@kitlangton kitlangton enabled auto-merge (squash) May 8, 2026 20:54
@kitlangton kitlangton disabled auto-merge May 8, 2026 20:56
@kitlangton kitlangton merged commit 5bb7b23 into dev May 8, 2026
10 checks passed
@kitlangton kitlangton deleted the llm-core-patch-api branch May 8, 2026 20:56
katosun2 pushed a commit to katosun2/opencode that referenced this pull request May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working contributor Vouched

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant